“Data is the new oil.” We often hear this phrase as a way to describe how data powers the modern economy. Today, companies have access to in-depth data on their customers, product mix, the market and much more in a way that simply wasn’t possible just a few decades ago. Data is now everywhere, and streaming in from all directions at higher volumes every day.
So how do you manage all these different data sources in a way that makes business sense? You’ve got to control the flow of this new oil, or you’ll drown in it without even getting it into the engine of your business.
This is where Data Integration comes in.
What is Data Integration?
Simply put, Data Integration is the practice of using Data Integration Architecture and Data Integration Tools to ensure consistent access to and delivery of data on the subjects that are relevant for your business. It’s about organizing your different data streams for the applications and processes of your company so that you can more effectively fuel your business with this important knowledge.
The landscape of vendors providing Data Integration tools has been quite fragmented. Different players often serve specific niches with a tool optimized for a special type of Data Integration. The end result of this fragmentation is that each of the different teams inside a company may use a different Data Integration tool. This leads to overlap, redundancy, and the lack of a clear strategy for leveraging metadata across the company.
But things are changing now, as companies realize they need to view Data Integration from a more holistic perspective. This is driving consolidation in the market, such that companies can now purchase complete Data Integration systems that address all their needs.
“While it’s true that organizations have started thinking more holistically about integrations, there is still a long way to go for many organizations,” says Janne Kärkkäinen, Chief Product Officer at integration service provider ONEiO. “Reaching higher levels of data maturity – where the entire organization can stand behind a decision to integrate all of their data sources – takes a considerable amount of time, as well as expert domain knowledge from all of the departments involved. These resources and knowledge are scarce, which is why many of the more mature organizations have looked to supplement their consolidated data-integration systems with other tools that help them solve their domain-specific problems.”
Why is Data Integration important?
Let’s take a closer look at why Data Integration is so important. There are four main reasons:
Data Integration improves collaboration and helps to unify systems
These days, almost every company function and individual needs access to data. The company is also producing new data all the time. The IT department needs a way to provide access to it all through a self-service model, and Data Integration is the way to do so. Data Integration is basically a method for improving collaboration around data by unifying your different systems.
Data Integration saves time and boosts efficiency
Proper Data Integration means saving time later when it comes to preparing the data for analysis and then action. Using the right Data Integration tools also saves time for the IT department. It all adds up to greater efficiency across the whole company, as well as reduced costs.
Data Integration reduces errors and rework
Without the right Data Integration architecture, employees must do a lot of manual work to gather, process, and report data. This leaves too much room for error. It also results in rework, with different employees performing the same task again and again. Adding the right Data Integration system eliminates these problems.
Data Integration delivers more valuable data
In the end, Data Integration all comes down to improving the quality of your data. As the data is centralized into a single system, it’s easier to identify any quality issues and implement improvements. It’s basically about strengthening the foundation of data upon which to build your business.
A look at different Data Integration techniques
There are several techniques that can be used to perform Data Integration. Let’s take a look at four of them.
ETL (Extract > Transform > Load)
ETL is the classic data pipeline that converts raw data into the target system via these three steps. The ETL process is characterized by a staging area for transforming the data before it’s loaded to the data warehouse and then analyzed. ETL works well for smaller data sets that need complex transformations. But as business moves to the cloud, data sets grow, and the demand for real-time analytics increases, Data Integration is increasingly moving towards the other three techniques described below.
ELT (Extract > Load > Transform)
ELT is not to be confused with ETL! The main difference between the two is that with ELT there is no staging area for the data. It’s immediately loaded and transformed into the system, which is usually cloud-based. ELT is useful for large data sets that need to be processed quickly. It’s based on an approach called ‘micro batch’ or ‘delta load’ whereby the system only loads data that has been modified since the last load.
Data streaming
As the term suggests, data streaming is about continuously moving data through the system – from its source to its target – instead of loading it in batches. With data streaming, you can deliver data to your platform of choice in an analysis-ready format. The challenge with this method is that synchronizing information is not easy without a staging area. Also, if one part of the system breaks down then everything else gets affected too.
Application integration
In application integration, data is moved and synced between separate applications that work together. Application integration is typically used when two operational systems – one for HR and one for finance, for example – need to use the same data. Consistency between data sets is key for effectively performing this kind of application integration.
Real-life examples of data integration
Data Integration for the IT department
IT departments often collect and manage information related to the various configurations of their IT infrastructure into a system known as a configuration management database or CMDB. While it started as a simple way to keep track of what software and hardware configurations have been put in place in hardware devices, it has become challenging to keep this information up to date.
To ensure that the database is always up to date, all systems that trigger a change into this data – whether it’s an automated response to a monitoring event, discovery by an asset inventory tool, or a change request created into an ITSM system – should automatically be reflected in the CMDB data.
By integrating the data of multiple ITSM systems, the same data can also be made available across these systems. It’s what we call eBonding. eBonding is actually a somewhat old-fashioned solution, but it is still leaned upon a lot in order to integrate ITSM tools towards customers and vendors. Most businesses we see have realized that eBonding slows down their progress, as it limits what changes and improvements they can do within their tools.
Read more about eBonding: Is eBonding a scalable solution? No, definitely not. Here's why.
Data Integration for the Human Resources function
For HR, the most important Data Integrations are typically between the systems used for time reporting, payroll, and employee profiles. If an employee has worked overtime, for example, that information needs to be available to payroll so the person can be paid accordingly. Vacations, absences, and bonus payouts also need to be recorded properly. For everything to work smoothly, all the HR systems need to be in sync.
Data Integration for the Finance function
Finance is typically interested in keeping bookkeeping information synced with Enterprise Resource Planning (ERP) systems, so that pricing information, orders and sales are properly accounted for. Discounts and special offers, for example, need to be recorded properly in both systems as they can have a big impact on stock levels, as well as on top-and bottom-line quarterly numbers.
Data Integration in Marketing
In recent years there has been a lot of focus on the better integration of the data used for marketing and the data used by sales teams. For example, when running a webshop it’s useful if the contact and order forms are integrated with the system recording leads and sales. In this way, the marketing team can more accurately calculate the return on investment of a specific campaign. This in turn translates into improved targeting of marketing investments for future campaigns.
Data Integration in Sales
An efficient sales process relies on being able to find prospects who are both willing and able to purchase your product or service. When salespeople engage potential customers with a thorough understanding of that customer’s needs, they are better able to tailor the offering. When these various data points are all collected under one tool – through Data Integration – it can provide the salesperson with a 360-degree view of the customer.
“You can start by making basic customer information, such as annual revenues and number of employees, visible in your organization’s CRM system,” says Janne Kärkkäinen from ONEiO.
“While fields for these typically exist in most CRM systems, you can expand the customer’s profile by adding things like credit ratings, for example. This helps to ensure your salespeople are only talking to customers who they know can reliably pay for the products and services being purchased. Data Integration helps to keep all this information up to date, as it’s periodically and automatically retrieved from the finance system and brought into the CRM system.”
Data Integration in Procurement
When procuring goods and services from multiple suppliers, a lot of time can be wasted on accessing the various portals that suppliers offer. This fragmented supplier landscape also makes data collection difficult and can lead to sub-optimal pricing agreements. Bringing all your supplier info together in one place through Data Integration allows the whole organization to make smarter procurement decisions by obtaining bulk discounts or better purchasing terms.
Which Data Integration tools should you use?
Data integration tools vary in terms of the capabilities they offer and how they need to be utilized to get the most out of them. You should consider the following when choosing a data integration tool for your organization:
Where is your data located and how can it be reached?
Whether your data is on-premise or in the cloud can affect what tools you are able to use to integrate it. Not all Data Integration tools can access data that is stored on-premise, or vice versa. Reaching data that is stored on-premise may require configuration within the data integration tool in order to retrieve that data. Modern Data Integration tools run natively in the cloud to ensure that they are always up to date and you don’t need to worry about upgrading them. They are also able to communicate with databases and applications that are running on-premise.
How up-to-date do you want your data to be?
A traditional ETL approach to Data Integration may be well suited for you if the data you wish to integrate does not need to be immediately available in all of your systems. In those situations, Data Integrations can be scheduled to run periodically, for example as nightly, weekly, or monthly runs, or as ad-hoc runs upon request by the user needing up-to-date information.
Even if you do not necessarily need the data to be constantly updated, a real-time Data Integration does eliminate any questions around whether everyone is working with the latest data. In a growing number of scenarios, data needs to be always up to date and available. Make sure that the tool you are selecting is capable of either ELT, where you can shorten the intervals between data retrievals, or event streaming, where changes in the data trigger events that result in the data updates.
Does the tool support all of the transformations you may need to apply to it?
Databases and how we store data are not built equal, as there are various database schemas and technologies. Data can also be structured or unstructured. Perhaps a destination is not a database in the first place. All these possibilities require you to select a tool that can handle various transformations, mappings and translations before it can move data from a source to a destination.
Do you have the resources to maintain and further develop your data integrations?
The need to build Data Integrations stems from a business need for someone to have a unified view into a certain entity – whether it be a customer, an employee, your company’s financial position, or the assets it owns. The person who wants to gain this unified view is often not the person who can configure and maintain the integration. When selecting the tool for your organization, consider how it works with your current organizational structure and what resources you may need to hire to be able to support all of the integrations.
Some Data Integration tools are focused on the needs of integration developers, while some work really well for data scientists and analysts. Finally, there are tools to allow business users to easily connect the tools they are working with and the data sources from which they need to retrieve information. These modern tools don’t require you to learn how to code, and you don’t need to understand how exactly the data is structured. You also do not need to worry about whether the integration platform is doing what it was configured to do.
Does your Data Integration tool vendor understand your use cases?
This is probably the most important thing to consider when selecting a Data Integration tool vendor. Is the tool you are looking at aimed at solving your specific problems, or can it solve the problem you face? Check to see whether the tool vendor has solved similar use cases to what you are looking to solve. Also, consider how much support they can offer for your specific needs, and whether they are willing to work with you along every step of your Data Integration journey. More often than not, Data Integration projects are a challenge not due to the technology, but due to the lack of understanding of your specific context.
Shortcomings of Data Integration and when to use ONEiO
With the newer approaches to Data Integration, its traditional shortcomings are becoming less pronounced. Transferring data without taking into account whether it even needs to be moved is wasteful and in some cases impossible due to the volume of data.
Many Data Integration tools still focus on how to manage and transfer these masses of data in the most efficient way possible. This approach is insufficient, however, in environments where the information needs to be available in near real-time. If this is the case for you, choosing an integration platform that allows for direct application integrations ensures that once one system is updated, this change can be reflected at once in all other systems where the information is needed.
ONEiO’s Integration Automation Platform is ideal for the situation where you simply cannot wait for data to be synchronized and where processes are event-driven. ONEiO combines ELT, data streaming, and application integration approaches to deliver data to the various systems an organization utilizes while maintaining correct order and context even during a process that takes days to complete.
If you are looking for a way to keep your tools and people up to speed, contact us for a free 15-minute assessment to see how we can help you to integrate your data.
